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jl^  GAO 

^^^^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Accounting  Office 
Washington,  DC  20548 


February  1,  2002 

The  Honorable  Edward  M.  Kennedy 
Chairman,  Committee  on  Health,  Education, 

Labor  and  Pensions 
United  States  Senate 

The  Honorable  James  M.  Jeffords 
United  States  Senate 

The  Congress  passed  the  Workforce  Investment  Act  (WlA)  in  1998  to  begin 
unifying  a  fragmented  employment  and  training  system  and  to  better  serve 
job  seekers  and  employers.  To  create  a  more  comprehensive  workforce 
investment  system,  WlA  requires  states  and  localities  to  bring  together 
most  federally  funded  employment  and  training  services  into  a  single 
system,  called  the  one-stop  center  system.  Seventeen  programs  across 
four  federal  agencies — programs  such  as  the  Employment  Service 
(Wagner-Peyser)  and  Adult  Education  and  Literacy — must  provide 
services  through  this  one-stop  system.^  Three  of  these  programs,  whose 
funding  is  authorized  by  WIA  under  Title  1  to  provide  services  to  adults, 
dislocated  workers,  and  youth,  replace  those  previously  funded  under  the 
Job  Training  Partnership  Act  (JTPA).^  These  three  new  WlA  programs, 
authorized  at  about  $3.7  billion  in  fiscal  year  2001,  have  performance 
measures  established  under  WlA  that  states  and  localities  must  track  in 
order  to  demonstrate  the  effectiveness  of  the  programs.  These 
performance  measures  gauge  program  results  in  the  areas  of  job 
placement,  employment  retention,  and  earnings  change,  as  well  as  skill 
attainment  and  customer  satisfaction.  States  are  held  accountable  by  the 
U.S.  Department  of  Labor  for  their  performance  in  these  areas.  If  they  fail 
to  meet  their  expected  performance  levels,  they  may  suffer  financial 
sanctions;  if  they  meet  or  exceed  their  levels,  they  may  be  eligible  to 


hn  addition  to  the  Department  of  Labor,  which  oversees  the  implementation  of  WIA, 
programs  funded  through  the  Departments  of  Education,  Health  and  Human  Services,  and 
Housing  and  Urban  Development  are  required  to  provide  services  for  certain  of  their 
programs  through  this  one-stop  center  system. 

^While  WIA  reauthorizes  or  amends  other  existing  programs,  including  Adult  Education  and 
Literacy,  Employment  Service  (Wagner-Peyser),  and  Vocational  Rehabilitation,  the 
performance  measures  apply  only  to  the  adult,  dislocated  worker,  and  youth  programs 
under  Title  I,  which,  for  this  report,  we  will  refer  to  as  “WIA-funded  programs.” 
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receive  additional  funds,  called  incentive  grants.^  In  order  to  make 
objective  decisions  about  which  states  receive  incentives  and  sanctions, 
WIA  seeks  to  ensure  that  states  collect  and  report  comparable 
performance  data  on  all  participants. 

In  an  effort  to  assess  the  effectiveness  of  WIA’s  performance  measures 
and  whether  they  will  yield  useful  information  that  can  clearly 
demonstrate  performance  imder  WIA,  we  assessed  (1)  the  progress  states 
and  localities  have  made  and  the  issues  they  have  faced  in  implementing 
performance  measures  for  the  three  WIA-funded  programs;  (2)  how  useful 
the  WIA  performance  measures  are  in  accurately  gauging  the  performance 
of  the  three  WIA-funded  programs;  and  (3)  beyond  gauging  the 
performance  of  the  three  WIA-funded  programs,  how  well  the 
performance  of  the  one-stop  system  is  being  measured. 

Our  review  is  based  on  a  smvey  of  WIA  program  administrators  in  all  50 
states;  visits  to  Florida,  Indiana,  Missouri,  Oregon,  and  Peimsylvania, 
where  we  interviewed  state  and  local  workforce  investment  officials; 
interviews  with  Labor  officials  and  with  national  associations  representing 
state  and  local  workforce  development  officials;  and  a  review  of  relevant 
documents.  We  selected  the  five  states  for  site  visits  based  on  a  variety  of 
factors,  including  their  experience  using  unemployment  insurance  (UI) 
wage  record  data  to  measure  outcomes,  the  status  of  their  data  collection 
systems,  and  their  progress  in  developing  additional  measures  for  state 
and  local  use.  Our  review  focused  on  measures  for  Title  I  Adult,  Dislocated 
Worker,  and  Older  Youth  programs.  We  did  not  review  the  performance 
measures  for  younger  youth  (age  14-18).'‘  We  conducted  our  work  between 
December  2000  and  August  2001  in  accordance  with  generally  accepted 
government  auditing  standards. 


Results  in  Brief 


States  and  localities  have  taken  action  to  implement  the  new  performance 
measurement  system  for  the  three  WIA-funded  programs  but  reported 
confronting  a  number  of  challenges  in  doing  so.  As  part  of  their 


^States  must  also  meet  performance  levels  for  programs  authorized  under  the  Adult 
Education  and  Family  Literacy  Act  and  Carl  D.  Perkins  Vocational  and  Applied  Technology 
Education  Act  (Perkins)  to  be  eUgible  for  incentive  grants. 

^We  also  did  not  review  measures  for  eligible  training  providers,  which  were  reviewed  in 
our  study,  U.S.  General  Accounting  Office,  Workforce  Investment  Act:  Better  Guidance 
Needed  to  Address  Concerns  Over  New  Requirements,  GAO-02-72  (Washington,  D.C. : 
2001). 
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implementation  efforts,  states  had  to  change  the  way  they  collected  and 
reported  performance  data  from  the  way  they  were  collected  for  previous 
programs.  Most  states  decided  to  develop  new  automated  data  systems, 
but  15  states  do  not  have  their  systems  completely  in  place.  The  timing  of 
the  guidance  from  Labor  on  how  to  report  the  data-not  issued  until  well 
into  the  first  year  of  implementation-delayed  the  development  of  these 
systems.  States  and  localities  also  faced  challenges  in  implementing  these 
measures  due  to  their  complexity  and  the  resource  demands  created  by 
new  measures;  some  had  to  develop  new  procedures  to  obtain  access  to 
sensitive  records.  For  example,  a  new  measure  to  assess  customer 
satisfaction  for  job  seekers  and  employers  requires  states  to  implement 
complicated  procedures  to  conduct  a  customer  satisfaction  survey.  In 
addition,  state  officials  expressed  concern  that  the  levels  of  performance 
that  they  are  expected  to  achieve  are  too  high.  Because  the  program  was 
in  its  first  year,  there  were  little  or  no  data  that  could  be  used  as  a  point  of 
comparison,  or  baseline,  for  establishing  performance  levels.  Furthermore, 
state  and  local  officials  believed  that  the  established  performance  levels 
did  not  take  into  account  local  economic  conditions.  The  performance 
levels  are  of  particular  concern  to  state  and  local  officials  because  failure 
to  meet  them  can  result  in  financial  sanctions.  As  a  result,  states  may  be 
choosing  to  serve  only  those  job  seekers  who  are  most  likely  to  be 
successful.  For  example,  to  meet  their  performance  levels  to  replace 
earnings  for  workers  who  have  been  dislocated  from  their  jobs,  local  staff 
might  provide  WIA  services  only  to  job  seekers  who  have  the  best 
prospect  of  finding  work  that  can  match  or  surpass  their  previous 
earnings. 

Even  when  fully  implemented,  WIA  performance  measures  may  still  not 
provide  a  true  picture  of  WIA-funded  program  performance  largely 
because  data  are  not  comparable  across  states  or  timely.  The  measures 
include  many  of  the  indicators  relevant  to  an  employment  and  training 
program,  such  as  getting  and  keeping  jobs  and  increasing  wages  and  skills. 
However,  data  may  not  accurately  gauge  performance  and  are  not 
comparable  across  states  for  a  variety  of  reasons.  For  example.  Labor’s 
guidance  on  enrolling  job  seekers — a  process  called  registration — does 
not  provide  clear  direction  on  when  to  start  collecting  performance  data 
on  participants,  and  we  found  that  states  and  localities  differed  on  whom 
they  tracked  and  when.  In  addition,  UI  wage  records,  required  in  order  to 
track  performance  outcomes,  are  not  easily  used  to  identify  employment 
across  state  lines,  and  some  states  are  more  successful  than  others  in 
obtaining  this  information.  Only  seven  states  are  currently  using  a 
clearinghouse — called  the  Wage  Record  Interchange  System  (WRIS) — that 
allows  states  to  exchange  Ul  data.  Many  of  the  remaining  states  are 
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reluctant  to  participate,  often  citing  cost  concerns.  In  addition,  the  time  lag 
associated  with  UI  data  becoming  available  (as  much  as  9  months  or 
longer)  results  in  states  not  being  able  to  use  the  data  to  manage  their 
programs  in  the  short  term. 

While  there  are  measures  designed  to  gauge  the  performance  of  the  three 
WIA-funded  programs,  there  are  no  measures  to  gauge  the  performance  of 
the  one-stop  system  as  a  whole.  At  least  17  programs  provide  services 
through  the  one-stop  system  and  most  have  their  own  performance 
measures.  Although  these  performance  measures  may  be  used  for 
assessing  outcomes  for  individual  programs,  they  cannot  be  used  to 
measure  the  success  of  the  overall  system.  For  example,  no  program  has  a 
measure  to  track  job  seekers  who  use  only  self-service  or  informational 
activities  offered  through  the  one-stop,  which  may  constitute  a  large 
proportion  of  job  seekers.  Not  knowing  how  many  job  seekers  use  the 
one-stop’s  services  limits  the  one-stop’s  ability  to  assess  its  impact. 
Furthermore,  state  and  local  officials  told  us  that  having  multiple 
performance  measures  has  impeded  coordination  among  programs.  There 
has  been  limited  progress  in  developing  overall  performance  measures  for 
the  one-stop  system.  Labor  convened  a  working  group  to  develop 
additional  indicators  of  the  one-stop  system’s  performance,  but  they  have 
not  yet  issued  them. 

To  provide  the  Congress  and  the  public  with  a  more  accurate  picture  of 
WIA  performance,  we  are  making  recommendations  to  Labor  in  this  report 
to  postpone  the  implementation  of  financial  sanctions,  to  expedite  the 
development  of  criteria  for  renegotiating  performance  levels,  to  more 
clearly  define  policies  and  measures,  and  to  develop  ways  to  address  the 
challenges  in  using  UI  data.  In  its  written  comments.  Labor  concurred 
with  our  findings  and  recommendations. 


Background 


Labor  required  states  to  implement  major  provisions  of  WIA  Title  I  by  July 
1,  2000,  although  some  states  began  implementing  provisions  of  WIA  as 
early  as  July  1999.  Services  provided  under  WIA  represent  a  marked 
change  from  those  provided  under  the  previous  program,  allowing  for  a 
greater  array  of  services  to  the  general  public.  WIA  is  designed  to  provide 
for  greater  accountability  over  what  existed  previously:  it  established  new 
performance  measures  and  a  new  requirement  to  use  UI  data  to  track  and 
report  on  the  achievements  of  the  three  WIA-funded  programs.  WIA  also 
requires  that  many  federal  programs  work  together  to  provide 
employment  and  training  services  through  the  one-stop  system. 
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WIA-Funded  Services  Program  services  provided  under  WIA  represent  a  marked  change  from 

Represent  a  Change  from  those  provided  under  JTPA.  When  WIA  was  enacted  in  1998,  it  replaced 
those  Funded  under  JTPA  JTPA  programs  for  economically  disadvantaged  adults  and  youth  and 

for  dislocated  workers  with  three  new  programs — WIA  Adult,  Dislocated 
Worker,  and  Youth — that  provide  a  broader  range  of  services  to  the 
general  public,  no  longer  using  income  to  determine  eligibility  for  all 
program  services.*^  The  newly  authorized  WIA  programs'^  no  longer  focus 
exclusively  on  training  but  provide  for  three  tiers,  or  levels,  of  service  for 
adults  and  dislocated  workers:  core,  intensive,  and  training.  Core  services 
include  basic  services  such  as  job  searches  and  labor  market  information. 
These  activities  may  be  self-service  or  require  some  staff  assistance. 
Intensive  services  include  such  activities  as  comprehensive  assessment 
and  case  management — activities  that  require  greater  staff  involvement. 
Training  services  include  such  activities  as  occupational  skills  or  on-the- 
job  training.  These  tiers  of  WlA-funded  services  are  provided  sequentially. 
That  is,  in  order  to  receive  intensive  services,  job  seekers  must  first 
receive  at  least  one  core  service;  to  receive  training  services,  a  job  seeker 
must  first  receive  at  least  one  core  service  and  then  at  least  one  intensive 
service.  Key  to  moving  from  core  to  a  higher  level  of  services  is  that  the 
services  are  needed  to  help  job  seekers  become  self-sufficient.  Labor’s 
guidance  provides  for  monitoring  and  tracking  to  begin  when  job  seekers 
receive  core  services  that  require  significant  staff  assistance.  Job  seekers 
who  receive  core  services  that  are  self-service  in  nature  are  not  included 
in  the  performance  measures. 


WIA  Performance 
Measures  Are  Designed  to 
Increase  Accountability  for 
Three  WIA-Funded 
Programs 


WIA  is  designed  to  provide  for  greater  accountability  than  the 
accountability  provided  for  under  JTPA.  It  does  so  by  establishing  new 
performance  measures  and  a  new  requirement  to  use  U1  data  to  track  and 
report  on  the  achievements  of  the  three  WIA-funded  programs.  According 
to  Labor,  performance  data  collected  from  the  states  in  support  of  the 
measures  are  intended  to  be  comparable  across  states  in  order  to  maintain 
objectivity  in  determining  incentives  and  sanctions.  They  are  also  intended 
to  provide  information  to  support  Labor’s  performance  goals  under  the 
Government  Performance  and  Results  Act  (GPRA)^  and  for  program 


^WIA’s  Youth  program  uses  low  income  as  an  eligibility  requirement. 

Authorized  through  fiscal  year  2003,  WIA’s  three  new  programs  had  a  budget  authority  of 
about  $3.7  billion  in  fiscal  year  2001. 

^GPRA  was  intended  to  focus  government  decisionmaking,  management,  and 
accountability  on  the  results  and  outcomes  achieved  by  federal  programs. 
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evaluation.  Some  of  the  measures  that  relate  to  adults,  dislocated  workers, 
and  older  youth  are  similar  to  those  used  under  JTPA,  including  job 
placement,  job  retention,  and  wage  gains  or  replacement.  Attainment  of  a 
credential** — a  degree  or  certification  of  skills  or  training  completed — and 
customer  satisfaction  for  both  job  seekers  and  employers  are  new  under 
WIA.  (See  table  1  for  a  complete  list  of  the  WIA  performance  measures  and 
appendix  1  for  a  more  complete  explanation  of  the  performance  measures 
discussed  in  this  report.) 


Table  1 :  Performance  Measures  for  the  Three  WIA-Funded  Programs 

WIA  funding  stream 

Performance  measure 

Adult 

1. 

Entered  employment  rate 

2. 

Employment  retention  rate  at  6  months 

3. 

Average  earnings  change  in  6  months 

4. 

Entered  employment  and  credential  rate 

Dislocated  worker 

5. 

Entered  employment  rate 

6. 

Employment  retention  rate  at  6  months 

7. 

Earnings  replacement  rate  in  6  months 

8. 

Entered  employment  and  credential  rate 

Older  youth 

9. 

Entered  employment  rate 

(age  19-21) 

10 

Employment  retention  rate  at  6  months 

11. 

Average  earnings  change  in  6  months 

12. 

Entered  employment/education/training  and 
credential  rate 

Younger  youth 

13. 

Skill  attainment  rate 

(age  14-18) 

14. 

Diploma  or  equivalent  attainment 

15. 

Placement  and  retention  rate 

Customer  satisfaction 

16. 

Customer  satisfaction  for  participants 

17. 

Customer  satisfaction  for  employers 

Source:  U.S.  Department  of  Labor  Employment  and  Training  Administration,  Training  and 

Empioyment  Guidance  Letter  No.  7-99  (Mar.  3,  2000). 

In  contrast  to  JTPA,  for  which  data  on  outcomes  were  obtained  through 
follow-ups  with  job  seekers,  WIA  requires  states  to  use  UI  wage  records  to 
track  outcomes.  According  to  Labor’s  guidance,  if  a  program  participant 
does  not  appear  in  the  Ul  wage  records,  states  may  use  supplemental  data 
sources,  such  as  follow-ups  with  participants  and  employers,  to  track 

entered  employment,  retention. 

and  credential  attainment.  However,  only 

Ul  wage  records  may  be  used  to  calculate  earnings  change  and 

replacement. 

^Guidance  from  Labor  defines  a  credential  as  a  nationally  recognized  degree  or  certificate 
or  a  recognized  state/locally  defined  credential. 
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Tinlike  JTPA,  which  established  expected  performance  levels  using  a 
computer  model,  WIA  requires  states  to  negotiate  with  Labor  to  establish 
expected  performance  levels  for  each  measure.  States,  in  turn,  must 
negotiate  performance  levels  with  each  local  area.  The  law  requires  that 
these  negotiations  take  into  account  differences  in  economic  conditions, 
participant  characteristics,  and  services  provided.  To  derive  equitable 
performance  levels.  Labor  and  the  states  use  historical  data  to  develop 
their  estimates  of  expected  performance  levels.  These  estimates  provide 
the  basis  for  negotiations. 

WIA  holds  states  accountable  for  achieving  their  performance  levels  by 
tying  those  levels  to  financial  sanctions  and  incentive  funding.  States  that 
meet  their  performance  levels  under  WIA  are  eligible  to  receive  incentive 
grants  that  may  generally  range  from  $750,000  to  $3  million.®  To  be  eligible 
for  an  incentive  grant,  states  must  also  meet  levels  established  under  the 
Department  of  Education’s  Vocational  Education  (Perkins  Act)  and  Adult 
Education  and  Literacy  programs.  States  that  do  not  meet  their 
performance  levels  under  WIA  are  subject  to  sanctions.^  If  a  state  fails  to 
meet  its  performance  levels  for  1  year.  Labor  provides  technical 
assistance,  if  requested.  If  a  state  fails  to  meet  its  performance  levels  for  2 
consecutive  years,  it  may  be  subject  to  up  to  a  5-percent  reduction  in  its 
annual  WIA  formula  grant.  Under  JTPA,  the  most  stringent  sanction  was 
the  possible  reorganization  of  the  local  service  delivery  organization. 


^According  to  Labor’s  guidance,  states  must  achieve  a  cumulative  score  of  100  percent  or 
more  in  each  program  area — adult,  dislocated  worker,  and  youth — and  for  customer 
satisfaction  and  at  least  80  percent  of  the  negotiated  performance  level  for  all  17  measures. 
Failing  to  achieve  the  80-percent  level  for  any  one  performance  measure  disqualifies  a  state 
from  being  considered  for  an  incentive  grant. 

According  to  Labor,  while  the  agency  has  sole  responsibility  for  administering  sanctions 
under  WIA,  awarding  incentives  is  a  joint  effort  involving  Labor  and  the  Department  of 
Education. 
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WIA  Requires  that  Many 
Federal  Programs  Work 
Together  to  Provide 
Services  through  the 
One-Stop  System 


In  addition  to  establishing  the  three  new  programs,  WIA  requires  that 
states  use  the  one-stop  center  system  to  provide  services  for  these  and 
many  other  employment  and  training  programs.  This  system  was 
developed  by  states  prior  to  WIA  through  One-Stop  Planning  and 
Implementation  Grants  from  Labor.  About  17  programs  funded  through 
four  federal  agencies  are  now  required  to  provide  services  through  the 
one-stop  center  under  WIA."  Table  2  shows  the  programs  that  WIA 
requires  to  provide  services  through  the  one-stop  centers  (termed 
mandatory  programs)  and  the  related  federal  agency. 


Table  2.  WlA’s  Mandatory  Programs  and  Related  Federal  Agencies 

Federal  agency 

Mandatory  programs 

Department  of  Labor 

WIA  adult 

WIA  dislocated  worker 

WIA  youth 

Employment  Service  (Wagner-Peyser) 

Trade  adjustment  assistance  programs 

Veterans’  employment  and  training  programs 
Unemployment  Insurance 

Job  Corps 

Welfare-to-Work  grant-funded  programs 

Senior  Community  Service  Employment  Program 
Employment  and  training  for  migrant  and 
seasonal  farm  workers 

Employment  and  training  for  Native  Americans 

Department  of  Education 

Vocational  Rehabilitation  Program 

Adult  Education  and  Literacy 

Vocational  Education  (Perkins  Act) 

Department  of  Health  and  Human 
Services 

Community  Services  Block  Grant 

Department  of  Housing  and  Urban 
Development  (HUD) 

HUD-administered  employment  and  training 

Under  WIA,  employers  are  expected  to  play  a  key  role  in  establishing 
regional  workforce  development  policies,  deciding  how  services  should  be 
provided  in  the  one-stop,  and  overseeing  one-stop  operations.  Employers, 
who  are  encouraged  to  use  the  one-stop  system  to  fill  their  job  vacancies, 
are  also  seen  as  key  one-stop  customers  under  WIA. 

"in  addition,  three  other  categories  of  programs  are  required  to  provide  services  through 
the  one-stop  center:  Youth  Opportunity  Grants;  demonstration,  pilot,  multiservice, 
research,  and  multistate  projects;  and  national  emergency  grants.  Because  they  are  of 
limited  scope,  we  did  not  include  them  in  our  total. 
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States  and  Localities 
Are  Taking  Action  to 
Implement  WIA 
Performance 
Measures  but  Face 
Challenges  in  Doing 
So 


States  and  localities  are  taking  action  to  implement  performance  measures 
for  the  three  WIA-funded  programs,  but  they  have  confronted  several 
challenges  in  doing  so.  To  implement  these  measures,  states  and  localities 
had  to  change  the  way  they  collected  and  reported  performance  data. 

Most  states  we  surveyed  had  to  create  new  automated  data  systems  to 
collect  and  report  WIA  data.  Many  state  systems,  however,  are  still  not 
completely  in  place.  The  lack  of  final  guidance  from  Labor  on  how  to 
report  the  data  slowed  the  development  of  these  systems.  States  and 
localities  also  faced  challenges  in  implementing  the  measures  due  to  their 
complexity  and  the  resource  demands  they  created,  and  some  had  to 
develop  new  procedures  to  obtain  U1  wage  records.  In  addition,  states 
faced  a  new  negotiation  process  with  Labor  to  set  performance  levels  for 
each  measure.  Many  states  believe  these  levels  are  too  high  because  little 
or  no  baseline  data  were  used,  and  the  negotiations  did  not  sufficiently 
account  for  differences  in  economic  conditions  and  populations  served. 
Under  WIA,  performance  levels  are  now  tied  to  incentives  and  sanctions 
so  that  states  can  be  financially  rewarded  if  they  meet  them  or  penalized  if 
they  do  not.  States  reported  that  the  need  to  meet  these  performance 
levels  may  lead  local  staff  to  focus  WIA-funded  services  on  job  seekers 
who  are  most  likely  to  succeed  in  their  job  search  or  who  are  most  able  to 
make  wage  gains. 


Management  Information 
Systems  Are  Still  Being 
Developed  to  Collect  and 
Report  WIA  Data 


As  part  of  implementing  WIA  performance  measures,  states  had  to  develop 
automated  data  systems  to  track  the  activities  of  individual  WIA 
participants  and  report  on  performance.^^  Based  on  our  survey,  most  states 
developed  a  new  automated  data  system,  or  management  information 
system  (MIS),  to  collect  and  report  WIA  performance  data  at  the  state 
level.  The  remaining  states  adapted  their  previous  data  collection  systems 
used  under  JTPA.  However,  15  states,  regardless  of  whether  they  were 
developing  a  new  system  or  adapting  their  existing  system,  reported  that, 
as  of  August  2001,  they  did  not  have  their  system  completely  in  place.  All 
states  expect  to  have  completed  their  systems  by  July  2002.  In  some  states, 
local  areas  do  not  use  the  state  MIS  to  collect  local  WIA  performance 
information.  In  these  states,  local  areas  must  develop  their  own  systems, 
taking  time  and  resources  to  do  so. 


^^WIA  requires  states  to  establish  and  operate  management  information  systems  based  on 
guidelines  established  by  Labor,  which  are  designed  to  promote  the  efficient  collection  of 
information. 
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The  lack  of  timely  reporting  guidance  slowed  the  development  of  the  data 
systems.  Final  guidance  on  how  states  must  report  their  performance  to 
Labor  was  issued  to  the  states  in  March  2001 — 8  months  after  the  states 
were  required  to  implement  major  provisions  of  WIA  and  begin  collecting 
data.“  Lack  of  final  guidance  resulted  in  delays  and  costly  program 
changes  as  states  and  local  areas  developed  and  adjusted  their  final 
systems.  For  example,  one  local  area  we  visited  decided  to  continue  using 
its  old  system  and  delayed  the  development  of  a  new  system  pending  final 
guidance  because  it  would  be  too  costly  and  time-consuming  to  develop  a 
system  that  might  need  to  be  changed.  All  states,  regardless  of  whether  or 
not  they  had  implemented  a  new  system,  had  to  make  changes  in  their 
automated  MIS  systems  to  accommodate  the  final  guidance. 


Complex  Performance 
Measures  Stymied  State 
and  Local  Efforts  to 
Implement  Them 


States  and  localities  reported  that  the  complexity  of  WIA’s  new 
performance  measures  made  them  difficult  and  time-consuming  to 
implement.  Many  of  the  states  that  we  surveyed  commented  that  the 
measures  were  hard  to  follow  because  the  calculations  for  the  measures 
are  complex  and  sometimes  confusing,  specifically  who  to  include  in  the 
measures,  when  to  collect  the  data  for  the  measures,  and  how  to  calculate 
the  measures. 


•  Knowing  who  to  inclnde  in  the  measnres.  It  is  difficult  to  know 
whether  or  not  a  job  seeker  should  be  counted  in  the  measure  for  a 
program,  even  if  he  or  she  is  served  by  the  program.  For  example,  a 
participant  in  the  adult  program  who  is  already  employed  must  be 
included  in  the  retention  and  wage  gain  measures  but  cannot  be 
counted  in  the  entered  employment  measure.  Yet,  for  the  dislocated 
worker  program,  the  entered  employment  measure  can  include  those 
who  may  be  employed  when  they  enter  the  program. 

•  Knowing  when  to  measnre  performance  for  participants.  The 

data  for  different  performance  measures  can  be  collected  in  different 


^^U.S.  Department  of  Labor  Employment  and  Training  Administration,  Training  and 
Employment  Guidance  Letter  No.  14-00  (Mar.  5,  2001). 

^^Calculations  for  the  measures  are  found  in  U.S.  Department  of  Labor  Employment  and 
Training  Administration,  Training  and  Employment  Guidance  Letter  (TEGL)  No.  7-99  (Mar. 
3,  2000). 

^^Dislocated  workers  may  include  those  who  have  obtained  interim  employment  after  being 
displaced. 
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quarters  of  the  program  year.  For  example,  for  customer  satisfaction, 
data  can  be  collected  at  two  points  in  time,  depending  on  how  the 
participant  exited  the  program  (see  fig.  1).  Data  on  entered 
employment  for  participants  in  the  adult  program  are  collected  in  the 
third  quarter  after  exit;  retention  data  are  collected  in  the  fifth  quarter 
after  exit.  “  Data  on  earnings  change  and  replacement  are  collected  at 
two  points  in  time:  pre-program  earnings  data  can  be  collected  at 
registration,  and  post-program  earnings  data  are  collected  in  the  fifth 
quarter  after  exit.^^  If  a  program  participant  does  not  appear  in  the  UI 
wage  records,  local  staff  can  collect  supplemental  data  to  establish 
employment  for  the  participant,  but  this  must  be  recorded  within  30 
days  after  the  WlA  participant  is  found  missing  from  the  wage  records. 
For  entered  employment,  staff  can  collect  supplemental  data  in  the 
fourth  quarter  after  a  participant  leaves  (or  exits)  the  program,  but  for 
retention,  staff  can  collect  it  in  the  sixth  quarter  after  exit. 
Supplemental  data  carmot  be  used  to  measure  earnings  change  and 
replacement.  Because  the  timing  of  data  collection  is  complex  and  can 
be  confusing,  one  local  area  in  Oregon  developed  a  tool  it  calls  the 
“bean  coimter”  to  help  local  staff  determine  when  to  follow-up  with 
participants  so  their  performance  coimts  in  the  calculations  (see  fig.  2). 


Figure  1 :  Timing  of  Data  Coiiection  by  Quarter  for  Seiected  Performance  Measures 


Program  Year  1999  Program  Year  2000 


Program  Year  2001 


Quarters  Quarter  4  Quarter  1  Quarters  Quarters  Quarter  4  Quarter  1  Quarters 


Measure 

1  Jan.-March 

April-June  I 

1  July-Sept. 

Oct. -Dec. 

Jan.-March 

April-June 

Participant 
registers  in 
WIA 

Participant 

exits 

WIA 

1  Customer 

1  satisfaction 

Collect 

data 

Collect 

data 

1  Entered 

1  emDiovment 

Collect  data 

1  Retention 

/ 

/ 

lEarnings  change/ 

1  reolacemenf 

/ 

SO  days 
supplemental 


to  get  I 
il  data  p 


Collect  data 


Collect  data 


Source:  Source:  U.S.  Department  of  Labor  TEGL  7-99. 


^^States  must  wait  two  full  quarters  after  exit  before  collecting  data  because  of  delays  in 
available  wage  records. 

^'^Data  for  pre-program  earnings  may  be  collected  at  registration  or  at  any  time  until  the 
fifth  quarter  after  exit. 
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Figure  2:  Oregon’s  Bean  Counter 


•  Knowing  how  to  calculate  the  measures.  In  order  to  calculate  the 
measures,  states  must  account  for  a  variety  of  factors.  The  type  and 
combination  of  these  factors  determine  the  calculations  that  will  be 
used.  For  example,  in  calculating  some  of  the  measures  for  the  adult 
program,  states  must  consider  (1)  whether  the  job  seeker  is  employed 
at  registration,  (2)  whether  he  or  she  is  employed  at  both  the  first  and 
third  quarters  after  exit,  and  (3)  whether  the  data  source  used  to 
confirm  employment  was  UI  records  or  supplemental  data.  This 
information  in  various  combinations  results  in  14  different  ways  that 
adult  participants  can  be  grouped  in  order  to  calculate  the  measures. 
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In  addition  to  noting  the  complexity  of  the  measures,  state  and  local 
officials  said  that  the  new  measures  taxed  their  resources.  States  had  to 
develop  procedures  to  collect  data  for  the  new  customer  satisfaction 
measure  in  compliance  with  detailed  guidance  from  Labor.  The  guidance 
calls  for  states  to  conduct  a  telephone  survey  from  a  random  sample  large 
enough  to  obtain  500  completed  surveys  from  both  participants  and 
employers.  Because  this  guidance  changed  over  time,  states  had  to  revise 
their  procedures  accordingly.  For  example,  a  revision  to  the  guidance 
issued  in  October  2001  required  states  to  maintain  an  up-to-date  list  of 
participants’  names  and  addresses  from  which  to  sample — a  requirement 
that  was  originally  voluntary. 

One  indication  of  states’  progress  in  implementing  these  measures  may  be 
reflected  in  their  ability  to  submit  complete  quarterly  reports. Quarterly 
reports  require  data  on  all  17  performance  measures.  For  the  quarterly 
report  that  was  due  in  May  2001,  all  states  submitted  their  reports,  but, 
according  to  Labor,  only  16  states  were  able  to  provide  data  for  all  17 
performance  measures.  For  the  quarterly  report  that  was  due  in  August 
2001 — more  than  1  year  after  WIA  implementation — all  states  submitted 
their  reports,  but  only  23  could  provide  data  on  employer  customer 
satisfaction. “  According  to  Labor,  states  could  not  fully  report  on 
customer  satisfaction  because  they  have  not  yet  fully  implemented 
procedures  to  measure  it.  One  state  had  to  compile  the  data  manually 
because  its  MIS  was  not  fully  operational. 


States  Adopt  New  WIA’s  new  requirement  that  states  use  UI  wage  records  to  measure 

Procedures  to  Access  outcomes  has  led  states  to  adopt  new  procedures  to  access  these  and 

Sensitive  Information  other  sensitive  records.  Unlike  JTPA,  which  relied  on  surveys  of 

participants  to  collect  information  on  employment  and  earnings,  WIA 
requires  UI  wage  records  to  be  used  as  the  primary  data  source  of 
employment  and  wage  information — and  the  only  data  source  for  some 
measures,  according  to  Labor’s  guidance.  To  obtain  employment  and 
earnings  information,  states  match  information  collected  on  individual 
WIA  participants  against  state  UI  wage  records.  To  access  UI  data  from 
the  state  agency  that  oversees  the  UI  database,  some  states  had  to 
establish  data-sharing  agreements.  In  Mississippi,  for  example,  the  agency 


authorized  by  WIA,  Labor  requires  states  to  submit  quarterly  summary  reports 
reflecting  the  state’s  performance  levels  and  other  activities. 

^^The  23  states  are  a  subset  of  the  53  states  and  territories  that  report  WIA  data  to  Labor. 
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responsible  for  overseeing  WIA — the  Mississippi  Development  Authority 
(MDA) — had  to  make  arrangements  with  the  agency  that  oversees  the  U1 
data — ^the  Mississippi  Employment  Security  Commission — to  have  them 
match  the  wage  records  and  provide  the  results  to  MDA.  In  addition,  some 
states  may  be  more  rigorous  in  protecting  the  confidentiality  of  U1  records 
through  privacy  laws,  which  may  add  obstacles  to  collecting  performance 
data.  For  example,  Oregon  law  prohibits  the  release  of  WIA  participants’ 
records  without  informed  consent.  Consequently,  program  providers  had 
to  enter  into  an  agreement  that  established  a  protocol  for  collecting  and 
sharing  the  data — one  that  developed  safeguards  to  protect  confidentiality. 
In  addition,  the  state  had  to  develop  a  process  to  ensure  that  WIA 
participants  consented  to  the  use  of  their  protected  records  in  this  way. 


States  Believe 
Performance  Levels  Are 
Set  Too  High 


All  the  states  we  visited  believed  that  some  of  the  established  performance 
levels  for  their  measures  were  set  too  high  for  them  to  meet — either 
because  they  were  set  in  absence  of  historical  or  baseline  data  or  because 
negotiations  did  not  sufficiently  account  for  variations  in  economic 
conditions  or  population  served.  States  reported  that  limitations  in 
available  baseline  data  made  it  difficult  to  set  fair,  realistic  performance 
levels.  The  new  measures  on  credentials  and  customer  satisfaction,  for 
instance,  had  no  prior  data  available  on  which  to  set  performance  levels. 
Where  baseline  data  were  available,  such  as  for  the  wage-related 
measures,  the  data  were  collected  under  JTPA,  a  program  whose  goals 
were  different  from  those  of  WIA.^"  In  addition,  some  states  believe  that 
the  performance  levels  did  not  account  for  variations  in  economic 
conditions,  such  as  the  slow  growth  in  new  or  existing  businesses  that 
some  areas  have  experienced.  Performance  levels  also  did  not  account  for 
the  many  economically  disadvantaged  or  hard-to-serve  individuals  seeking 
services  in  some  local  areas. 


States  Report  that 
Performance  Levels  May 
Determine  Who  Receives 
WIA-Funded  Services 


Many  states  reported  that  the  need  to  meet  performance  levels  may  be  the 
driving  factor  in  deciding  who  receives  WIA-funded  services  at  the  local 
level.  All  the  states  we  visited  told  us  that  local  areas  are  not  registering 
many  WIA  participants,  largely  attributing  the  low  number  of  WIA 
participants  to  concerns  by  local  staff  about  meeting  performance  levels. 


^°While  JTPA  focused  on  providing  training  to  the  economically  disadvantaged,  including 
the  hardest-to-serve,  WIA  provides  a  broader  range  of  services  to  all  individuals,  regardless 
of  their  eligibility  for  other  services. 
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Local  staff  are  reluctant  to  provide  WIA-funded  services  to  job  seekers 
who  may  be  less  likely  to  get  and  keep  a  job.  One  state  official  described 
how  local  areas  were  carefully  screening  potential  participants  and 
holding  meetings  to  decide  whether  to  register  them.  As  a  result, 
individuals  who  are  eligible  for  and  may  benefit  from  WIA-funded  services 
may  not  be  receiving  services  that  are  tracked  under  WIA. 

Performance  levels  for  the  measures  that  track  earnings  change  for  adults 
and  earnings  replacement  for  dislocated  workers  may  be  especially 
problematic.  Several  state  officials  reported  that  local  staff  were  reluctant 
to  register  already  employed  adults  or  dislocated  workers.  Officials  in  one 
state  reported  that  some  local  areas  had  not  yet  registered  any  dislocated 
workers.  State  and  local  officials  explained  that  it  would  be  hard  to 
increase  the  earnings  of  adults  who  are  already  employed  or  replace  the 
wages  of  dislocated  workers,  who  are  often  laid  off  from  high-  paying,  low- 
skilled  jobs  or  from  jobs  that  required  skills  that  are  now  obsolete.  In 
addition,  for  dislocated  workers,  employers  may  provide  severance  pay  or 
workers  might  work  overtime  prior  to  a  plant  closure,  increasing  these 
workers’  earnings  before  they  are  dislocated.  As  a  result,  many  dislocated 
workers  who  come  to  the  one-stop  center  have  earned  high  wages  just 
prior  to  being  dislocated,  making  it  hard  to  replace — let  alone  increase — 
their  earnings.  If  high  wages  are  earned  before  dislocation  and  lower 
wages  are  earned  after  job  placement  through  WIA,  the  wage  change  will 
be  negative,  depressing  the  wage  replacement  level.  As  a  result,  a  local 
area  may  not  meet  its  performance  level  for  this  measure,  discouraging 
service  to  those  who  may  need  it. 

A  hypothetical  example  involving  two  workers  dislocated  at  the  same  time 
illustrates  this  point  (see  table  3).  One  worker  is  a  sales  clerk  with  limited 
skills  earning  $25,000,  the  other  a  long-time  factory  worker  with  obsolete 
skills  earning  $60,000.  Both  are  laid  off  from  work  and  go  to  their  local 
one-stop  center  seeking  job  placement  assistance.  The  clerk  is  placed  in  a 
new  job  as  a  receptionist  paying  $25,000.  By  calculating  his  wage 
replacement  from  his  salary  as  a  clerk,  the  one-stop  can  claim  a  wage 
replacement  rate  of  100  percent.  The  factory  worker  eventually  gets  a  job 
as  a  security  guard  earning  $30,000,  netting  a  wage  replacement  rate  of  50 
percent.  As  this  example  shows,  a  one-stop  center  can  meet  its 
performance  levels  more  easily  by  serving  the  clerk  than  by  serving  the 
factory  worker  even  though  both  job  seekers  may  need  the  one-stop 
system’s  resources  to  find  a  job  or  enhance  their  skills. 
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Table  3:  Comparing  Wage  Replacement  Rates  for  Two  Job  Seekers 

Scenario  1 :  Sales  Clerk 

Scenario  2:  Factory  Worker 

Sales  clerk  earning  $25,000 

Factory  worker  earning  $60,000  after  20 
years  of  employment 

•  Gets  laid  off 

•  Gets  laid  off 

•  Cannot  find  a  job 

•  Cannot  find  a  job 

•  Comes  to  one-stop  center  for  job 
placement  assistance 

•  Comes  to  one-stop  center  for  job 
placement  assistance 

•  The  center  gets  him  placed  as  a 
receptionist  making  $25,000 

•  The  center  gets  him  placed  as  a 
security  guard  earning  $30,000 

Wane  replacement  rate:  $25,000  =  100% 
$25,000 

Wane  replacement  rate:  $30,000  =  50% 
$60,000 

Note:  Example  uses  annual  full-time  earnings.  When  replacement  rates  are  actually  calculated  using 
Ul  wage  records,  they  will  be  based  on  earnings  reported  on  a  quarterly  basis. 


Some  states  and  Labor  are  making  efforts  to  address  this  disincentive  to 
serve  certain  job  seekers.  Indiana  instituted  a  policy  allowing  local  areas 
to  adjust  their  dislocated  worker  wage  replacement  rate  in  light  of  the 
significant  dislocations  they  are  facing.^^  Texas  uses  a  regression  model  to 
establish  local  performance  levels  that  adjust  for  differences  in  factors, 
such  as  economic  conditions  and  the  characteristics  of  individuals  served. 
Without  this  policy,  said  a  Texas  official,  WIA  programs  would  have 
registered  fewer  workers.  Similarly,  Michigan  substantially  reduced  the 
penalties  to  local  areas  for  failing  to  meet  performance  levels  and  found 
that  the  number  of  registered  participants  increased  as  a  result  of 
instituting  less  threatening  sanctions.  WIA  requires  that  states  be  allowed 
to  renegotiate  their  performance  levels  based  on  unanticipated 
circumstances.  Labor  is  currently  developing  criteria  that  states  can  use  to 
renegotiate  their  performance  levels  based  on  unanticipated 
circumstances,  such  as  changes  in  economic  conditions  due  to  plant 
closings  or  shifts  in  unemployment  for  the  current  and  future  years.  The 
guidance  is  expected  to  be  released  soon. 


^^The  adjusted  levels  would  apply  only  to  the  dislocated  workers  from  the  company  that 
has  been  affected,  not  to  the  entire  local  area. 
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Performance 
Measures  May  Not 
Accurately  Assess 
Performance  of  the 
Three  WIA-Funded 
Programs 


Even  when  fully  implemented,  WIA  performance  measures  may  still  not 
provide  an  accurate  picture  of  performance  for  the  three  WlA-funded 
programs  largely  because  data  are  neither  comparable  across  states  nor 
timely.  State  and  local  officials  generally  supported  many  of  the 
performance  measures  as  relevant  indicators  of  the  success  of  an 
employment  and  training  program.  However,  the  performance  data 
collected  and  reported  by  states  and  localities  are  not  comparable  largely 
because  of  the  lack  of  clear  guidance  on  when  to  collect  and  report 
performance  data  and  what  constitutes  a  credential — the  attainment  of  a 
certified  skill  or  degree.  In  addition,  while  U1  wage  records  are  one  of  the 
best  available  sources  of  employment  and  earnings  data,  limitations  in  the 
data  may  hinder  the  ability  of  states  and  local  areas  to  meet  their 
performance  levels  and  use  the  measures  for  short-term  program 
management. 


Performance  Measures 
Include  Relevant 
Indicators,  but  Lack  of 
Clear  Guidance  Affects 
Accuracy  and 
Comparability  of 
Performance  Data 


State  and  local  officials  in  the  states  we  visited  generally  support  many  of 
the  performance  measures  as  relevant  indicators  of  the  success  of  an 
employment  and  training  program.  For  example,  several  officials  cited  the 
wage-related  measures,  such  as  job  placement,  retention,  and  earnings 
change,  as  important  indicators  of  a  successful  employment  and  training 
program.  The  measures  are  also  generally  consistent  with  the  goal  of  WIA 
to  help  individuals  get  and  keep  jobs  and  increase  their  wages  and  skills. 
In  addition,  the  states  noted  that  the  measures  provide  a  good  basis  for 
long-term  evaluation. 

However,  the  performance  data  collected  and  reported  by  states  and 
localities  are  not  comparable — a  critical  component  in  creating  a  level 
playing  field  from  which  states’  relative  performance  can  be  evaluated. 
While  there  are  various  reasons  that  performance  data  are  not 
comparable,  one  of  the  chief  reasons  is  the  lack  of  clear  guidance  for 
collecting  and  reporting  performance  data  on  participants.  Labor  has 
provided  detailed  written  guidance  to  states  on  who  should  be  registered 
under  WIA  and  when  this  registration  should  occur,  but  the  guidance  is 
open  to  interpretation  in  some  areas.^^  The  lack  of  a  uniform 
understanding  of  when  registration  occurs  and  thus  who  should  be 
counted  toward  the  measures  raises  questions  about  both  the  accuracy 
and  comparability  of  states’  performance  data.  For  example,  the  guidance 
tells  states  to  register  adults  and  dislocated  workers  who  need  significant 


^^TEGL  No.  7-99. 
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staff  assistance  designed  to  help  with  job  seeking  or  acquiring 
occupational  skills,  but  the  state  can  decide  what  constitutes  significant 
staff  assistance.^^  The  guidance  provides  examples  of  when  to  register  job 
seekers,  but  it  sometimes  requires  staff  to  make  subtle  and  subjective 
distinctions.  For  example,  those  who  receive  initial  assessment  of  skill 
levels  and  the  need  for  supportive  services  are  not  to  be  registered;  those 
requiring  comprehensive  assessment  or  staff-assisted  job  search  and 
placement  assistance,  including  career  counseling,  are  to  be  registered.  In 
another  example  involving  the  classification  of  workshops,  job  seekers 
who  participate  are  to  be  registered  in  some  cases,  but  not  in  others. 

Labor  has  allowed  states  and  local  areas  flexibility  in  implementing  the 
registration  policy,  and  we  found  that  local  areas  differed  on  when  they 
registered  WIA  job  seekers.  In  one  local  area  we  visited,  the  one-stop 
center  registers  most  job  seekers  who  come  into  the  center,  even  if  staff 
assistance  is  minimal.  At  this  center,  a  general  orientation  is  sufficient  for 
the  job  seeker  to  be  registered  under  WIA.  In  contrast,  another  center  in 
the  same  state  registers  only  those  job  seekers  who  require  significant 
staff  assistance  and  are  likely  to  benefit  from  intensive  services.  Similar 
disparities  occurred  in  other  states  we  visited.  Labor  has  said  there  is  little 
consistency  across  states  in  registering  participants  and  has  convened  a 
work  group  to  develop  additional  guidance  on  registration,  but  as  yet,  the 
issues  remain  unresolved. 

The  lack  of  a  definition  for  the  credential  measure  is  also  leading  to 
performance  data  that  are  not  accurate  or  comparable  across  states.  Labor 
allows  the  states  and  local  areas  to  determine  what  constitutes  a 
credential  and  to  develop  a  statewide  list  of  approved  credentials  with 
input  from  employers.  Because  states  and,  in  many  cases,  local  areas  must 
define  what  constitutes  a  credential,  what  is  currently  counted  as  a 
credential  differs  within  and  across  states.  Some  states  may  strictly  define 
credentials  to  include  only  diplomas  from  accredited  institutions  or  use 
only  formal  training  completion  criteria  as  defined  by  education  partners. 
Other  states  may  expand  their  criteria  to  count  a  broad  variety  of 
credentials,  such  as  job  readiness,  on-the-job  experience,  and  completion 
of  workshops.  Labor  officials  note  that  states’  performance  levels  for  the 
credential  measure  are  negotiated  to  take  state  and  local  definitions  into 
account,  and  the  measure  is  intended  to  help  local  employers  gauge  the 
readiness  and  skill  level  of  job  seekers.  Nevertheless,  given  the  broad 


^^All  youth  who  receive  WIA-funded  services  are  required  to  be  registered. 
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range  of  definitions  states  and  localities  employ,  the  outcomes  on  the 
credential  measure  may  be  of  limited  value,  even  within  a  single  state. 


Challenges  in  Using  UI 
Wage  Record  Data  Also 
May  Affect  the  Accuracy 
and  Comparability  of 
Performance  Data 


UI  wage  records  are  one  of  the  best  available  data  sources  for  tracking  the 
employment  and  earnings  of  individuals — a  significant  improvement  over 
the  less  objective  self-reporting  methods  of  JTPA — but  the  limitations  of 
the  database  pose  challenges  that  need  to  be  addressed.  These  challenges, 
if  unresolved,  may  hinder  states’  ability  to  meet  their  performance  levels. 
As  we  have  reported  in  prior  work,^^  one  such  limitation  is  that  UI  wage 
records,  while  covering  about  94  percent  of  workers,  exclude  certain 
employment  categories,  such  as  self-employed  persons,  most  independent 
contractors,  military  personnel,  federal  government  workers,  and  postal 
workers.  States,  therefore,  must  develop  alternative  methods  to  track  WIA 
participants  who  are  employed  in  these  imcovered  occupations. 
Permsylvania,  for  example,  developed  a  partnership  with  other  states  in  its 
region  to  share  the  cost  of  purchasing  the  rights  to  federal  civil  service  and 
military  persoimel  data.  And  Florida  has  developed  agreements  with  the 
Department  of  Defense,  the  Office  of  Persoimel  Management,  and  the  U.S. 
Postal  Service  to  access  employment  and  wage  information  on  an  annual 
basis.  Our  survey  data  indicate  that  33  states  are  using  additional  or 
supplemental  data  to  compensate  for  uncovered  occupations,  with  only  27 
of  those  using  the  supplemental  data  to  count  towards  their  performance 
levels.  Thus,  at  least  23  states  have  not  used  additional  data  to  help  them 
meet  their  performance  levels. 


Another  limitation  is  that  state  UI  databases  include  only  wage  record 
information  on  job  seekers  who  get  jobs  within  their  state;  they  do  not 
track  job  seekers  who  find  jobs  in  other  states.  States  cannot  readily 
access  UI  wage  records  from  other  states  to  track  outcomes  under  WIA, 
making  it  difficult  to  track  individuals  who  receive  services  in  one  state 
but  get  a  job  in  another.  Over  one-third  of  all  of  the  states  we  surveyed 
reported  that  an  estimated  16  to  30  percent  of  cases  are  not  being  picked 
up  by  their  state’s  UI  wage  record  system.  To  fill  in  these  gaps,  seven 
states  have  agreements  with  other  states — often  those  that  share  a 
common  border — to  exchange  UI  information.  Indiana,  for  example, 
estabhshed  an  agreement  with  Illinois  to  trade  data.  If  data  are  missing  on 


^‘'See  U.S.  General  Accounting  Office,  Veterans’  Employment  and  Training  Service: 
Proposed  Performance  Measurement  System  Improved,  But  Further  Changes  Needed, 
GAO-01-580  (Washington,  D.C.:  2001). 
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particular  participants,  Illinois  sends  the  cases  to  Indiana  to  see  if  the 
Indiana  UI  wage  records  have  information  on  the  job  seeker.  The  value  of 
these  agreements,  however,  may  be  limited  because  job  seekers  may  find 
work  in  a  state  that  does  not  have  an  agreement  with  the  one  in  which 
they  received  services. 

Another  way  to  obtain  UI  data  on  workers  who  are  employed  out  of  state 
is  through  WRIS,  a  clearinghouse  that  makes  UI  wage  records  available  to 
states  seeking  employment  and  wage  information  on  their  WIA 
participants.  This  information  can  provide  outcome  data  on  WIA 
participants  to  help  states  meet  their  performance  levels.  While  WRIS  was 
available  for  states  to  use  by  July  2001,  only  7  out  of  the  50  states  are 
currently  able  and  ready  to  participate,  with  8  others  in  various  stages  of 
completing  the  requirements  for  participation.^'^  Although  many  states  have 
shown  an  interest  in  a  system  such  as  WRIS,  many  are  reluctant  to 
participate  because  Labor,  while  agreeing  to  cover  all  the  costs  of 
operating  WRIS  for  its  first  year,  has  not  yet  agreed  to  pay  for  subsequent 
years.  The  estimated  total  cost  of  operating  WRIS  is  $2  million  annually, 
but  states  have  not  been  given  a  definitive  answer  about  how  much  it 
would  cost  them  to  participate  after  this  first  year  if  Labor  does  not 
continue  funding.'^"  Because  of  this  uncertainty  regarding  future  costs, 
states  are  hesitant  to  commit  to  participation  in  WRIS.  If  not  all  states 
participate,  the  value  of  WRIS  will  be  diminished — even  for  participating 
states — because  no  data  will  be  available  from  nonparticipating  states’  UI 
wage  records. 


Lack  of  Timely  Data  Limits 
Use  of  Performance 
Measures  for  Short-Term 
Program  Management 


The  lack  of  timely  data,  due  to  the  time  lag  in  obtaining  UI  wage  records, 
makes  it  difficult  for  state  and  local  officials  to  use  the  performance 
measures  for  short-term  program  management — because,  for  the  wage- 
related  measures,  current  available  data  on  the  measures  will  reflect 
performance  from  the  previous  program  year.  While  UI  wage  records  are 
the  best  available  data  source  for  documenting  employment,  the  data 
collection  and  reporting  process  is  slow  and  time-consuming.  Data  are 
generally  collected  from  employers  only  once  every  quarter,  and 


participate,  states  are  required  to  be  technically  and  administratively  ready.  For 
example,  states  must  be  able  to  respond  to  queries  for  UI  wage  data  and  establish  internal 
security  measures  to  ensure  the  confidentiality  of  data. 

national  association  is  currently  working  a  contractor  to  study  the  costs  associated  with 
WRIS. 
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employers  have  30  days  after  the  quarter  ends  to  report  the  data.  In  many 
states,  employers — especially  small  businesses — are  allowed  to  submit 
data  in  paper  format,  which  then  must  be  converted  to  electronic  media. 
After  data  entry,  information  must  be  checked  for  errors  and  corrected.  All 
of  these  steps  take  time.  As  a  result,  WIA  program  administrators  are 
unable  to  get  a  timely  picture  of  program  performance.  For  example,  we 
asked  states  in  our  survey  how  quickly  job  placement  outcome  data  would 
be  available  to  them  from  UI  wage  records.  On  the  basis  of  our  survey,  we 
found  that  for  30  states,  the  earliest  time  period  that  job  placement  data 
would  be  available  is  6  months  or  more  after  an  individual  entered 
employment,  with  15  states  reporting  that  it  may  take  9  months  or  longer. 
Similarly,  for  the  employment  retention  measure,  over  half  of  states  report 
that  obtaining  this  information  could  take  a  year  or  longer.  (See  fig.  3.) 


Figure  3:  Number  of  Months  it  Takes  to  Receive  UI  Data  for  Job  Piacement 


Number  of  months 


Shortest  time  period  to  receive  UI  job  piacement  data 


Longest  time  period  to  receive  UI  job  placement  data 


Note:  Our  survey  asked  Iwo  questions  regarding  Ui  time  frames.  One  question  asked  for  the  eariiest 
date  states  wouid  expect  to  receive  Ui  data;  the  other  asked  for  the  iatest  date. 

Source:  GAO  survey. 
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The  time  delay  in  receiving  UI  wage  record  data  makes  it  difficult  for  state 
and  local  officials  to  use  the  performance  measures  to  gauge  the 
effectiveness  of  their  services.  States  report  that  not  being  able  to  get 
performance  results  in  the  same  program  year  is  a  problem:  it  makes  it 
difficult  to  manage  programs  and  improve  one-stop  services.  Labor  reports 
that  the  performance  measures  are  not  intended  to  be  a  management  tool. 
State  and  local  officials,  therefore,  must  develop  alternative  methods  if 
they  want  to  assess  the  quality  of  their  services  so  they  can  identify 
problems  and  improve  programs  in  a  timely  way.  Labor  has  encouraged 
these  efforts,  but,  while  some  local  areas  are  finding  ways  to  collect  data 
to  help  them  manage  their  programs,  there  is  no  cohesive  effort  at  the 
federal  level  to  share  strategies  and  promising  approaches  for  the  Adult 
and  Dislocated  Worker  Programs.^^ 


Existing  Performance 
Measures  Fail  to 
Gauge  Overall  One- 
Stop  Performance 


Although  there  are  performance  measures  for  the  three  WIA-funded 
programs  and  most  of  the  programs  required  to  provide  services  in  the 
one-stop,  no  measures  exist  to  assess  how  well  the  overall  one-stop 
systems  are  working.  The  success  of  the  one-stop  system  as  a  whole  is  not 
captured  by  the  program  measures  of  individual  one-stop  partners. 
Furthermore,  combining  the  performance  measures  from  mandatory 
programs  does  not  provide  a  comprehensive  picture  of  one-stop 
performance.  Even  when  measures  appear  the  same,  comparing  them  is 
difficult  because  of  differences  in  definitions  and  calculations.  Beyond 
failing  to  provide  a  complete  evaluation  of  one-stop  performance,  state 
officials  reported  that  the  separate  reporting  requirements  of  the  partner 
programs  have  hampered  coordination  within  their  one-stop  systems. 
While  WIA  did  not  establish  any  comprehensive  measures  to  assess  the 
overall  one-stop  system,  it  required  that  Labor  take  the  lead  in  developing 
optional  measures  to  help  states  assess  progress  toward  their  workforce 
investment  goals.  Labor  has  made  limited  progress  on  such  performance 
measures,  and  only  a  few  states  have  developed  their  own  overall 
measures. 


^Tabor’s  Office  of  Youth  Services  has  taken  action  to  help  youth  programs  in  this  area, 
such  as  creating  a  set  of  short-term  performance  measures. 
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Existing  Performance 
Measures  Fail  to  Capture 
Important  One-Stop 
Features 


The  existing  performance  measures  for  participating  one-stop  programs 
fail  to  capture  important  one-stop  features.  First,  it  is  difficult  to  get  an 
unduplicated  count  of  job  seekers  using  the  one-stop.  While  an  individual 
may  have  multiple  outcome  measures  for  the  services  received  through 
each  of  the  programs  at  a  one-stop,  there  is  no  single  outcome  measure  for 
multiple  services.  In  addition,  separate  reporting  systems  for  each  of  the 
programs  make  it  difficult  to  disaggregate  data  and  track  an  individual’s 
outcome  for  those  receiving  multiple  services. 


Second,  other  important  aspects  of  one-stop  performance  are  not  included 
within  the  existing  measures.  Customer  satisfaction  measures  used  in 
support  of  WIA-funded  programs  and  the  Employment  Service  fail  to 
measure  how  job  seekers  and  employers  believe  they  are  being  served  by 
the  one-stop  system  as  a  whole.  Instead,  these  measures  show  satisfaction 
with  the  individual  programs.  Employer  satisfaction  is  important  under 
WIA  because  WIA  created  a  more  private-sector  driven  system.  Capturing 
customer  satisfaction  of  the  system  as  a  whole  would  reflect  whether  job 
seekers  are  successful  at  attaining  the  services  they  need  to  get  jobs  and 
would  assess  whether  employers  are  satisfied  with  job  applicants  sent  to 
them  from  different  one-stop  programs. 


Finally,  state  and  local  officials  expressed  concern  that  a  large  portion  of 
one-stop  participants  are  not  included  in  performance  measures.  Many  job 
seekers  use  self-service  and  informational  activities  but  they  are  not 
tracked  and  counted  in  any  program  measures.  While  staff  time  and 
resources  are  used  to  establish  and  maintain  self-service  resource  rooms 
and  web  sites,  job  seekers  who  use  only  these  services  will  not  be 
included  in  any  of  the  performance  measures.  Without  any  information  on 
individuals  who  use  self-service,  it  will  be  difficult  for  Labor  to  show  how 
effectively  one-stops  are  being  used. 


Multiple  Measures  Cannot 
Be  Used  to  Measure  the 
One-Stop  System’s 
Performance 


Performance  measures  for  different  programs  often  track  similar 
outcomes,  as  figure  4  shows.  However,  the  measures  cannot  be  combined 
to  obtain  an  overall  view  of  one-stop  performance.  Although  the  same 
terms  are  used  in  various  performance  measures,  their  definitions  are  not 
identical.  For  example,  while  WIA  older  and  younger  youth  programs 
define  youth  as  being  between  the  ages  of  14  and  21,  the  laws  governing 
Job  Corps  and  HUD’s  Youthbuild  define  youth  as  being  between  the  ages 
of  16  and  24.  Similarly,  the  definition  of  veterans  is  different  for  the 
Employment  Service  and  Veterans’  Employment  and  Training  Service 
program.  The  differences  in  definitions  mean  that  assessing  the  outcomes 
for  youth  or  veterans  by  combining  the  performance  measures  of 
individual  programs  within  the  one-stop  setting  would  be  difficult. 
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Figure  4:  Comparison  of  Performance  Measures  for  17  One-Stop  Programs 


"May  include  the  attainment  of  a  credential  or  placement  in  education  or  training. 

Two  measures  for  customer  satisfaction — one  for  employers,  one  for  job  seekers — are  used  for  all 
WlA-funded  programs. 

This  program  reports  by  submitting  individual  records  on  program  participants. 

“Based  on  proposed  regulations. 

'Unemployment  Insurance  does  not  have  participant  performance  measures. 

"SCSEP  is  required  to  develop  performance  measures,  which  are  expected  to  be  in  place  for  program 
year  2003,  according  to  a  Labor  official. 

"Grantees  for  this  program  are  required  to  use  the  core  indicators  of  performance  common  to  WIA 
adult  and  youth  programs. 
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'Individual  grantees  propose  performance  in  their  individual  grantee  plans,  which  Labor  must  approve. 
'Measures  must  include  either  placement  or  retention  and  may  include  both. 

^For  this  program,  states  must  report  on  at  least  one  measure  from  each  of  six  national  goals.  The  six 
national  goals  are  (1)  number  employed,  (2)  number  maintaining  90-day  retention,  (3)  number  making 
progress  toward  literacy/GED,  (4)  number  gaining  health  care  coverage  through  employment,  (5) 
increase  in  the  availability  and  affordability  of  essenfial  services,  and  (6)  increase  in  access  to 
community  services  and  resources  by  low-income  people.  States  may  select  from  among  nafional 
measures  or  develop  their  own. 

Source:  Based  on  agency  regulations  and  documents. 

Besides  variations  in  definitions,  there  are  also  variations  in  how  measures 
are  calculated  in  different  programs.  For  example,  while  the  entered 
employment  rate  for  WlA’s  adult  program  is  defined  as  the  percentage  of 
workers  who  get  a  job  by  the  end  of  the  first  quarter  after  exit,  the  entered 
employment  rate  for  the  Employment  Service  is  defined  as  the  percentage 
of  workers  who  get  a  job  or  changed  employers  in  the  first  or  second 
quarter  after  registration.  As  a  result,  performance  data  from  these 
separate  programs  cannot  be  combined  to  yield  a  single  overall  score  to 
assess  various  performance  outcomes  of  the  one-stop  system.  The  Office 
of  Management  and  Budget  (0MB)  convened  a  work  group  representing 
federal  WIA  partners  to  look  at  common  definitions  and  measures  across 
programs,  which  may  issue  guidance  to  states  and  localities  on  WIA 
performance  measures  and  federal  requirements. 


Separate  Measures  May  Beyond  failing  to  provide  an  overall  picture  of  one-stop  performance,  state 
Impede  Coordination  officials  reported  that  separate  performance  measures  impede  the 

cooperation  of  one-stop  partners.^®  As  a  result,  even  though  WIA  was 
meant  to  establish  a  more  coordinated  workforce  development  system 
through  the  use  of  one-stops,  over  one-third  of  states  surveyed  expressed 
concern  that  individual  program  performance  measures  may  impede  this 
process.  Some  states  even  believed  that  separate  measures  caused 
competition  among  programs.  They  said  that  if  staff  did  not  understand 
that  a  participant  could  be  counted  in  more  than  one  program,  they  might 
not  direct  them  to  other  one-stop  programs.  For  example,  one  state 
reported  in  its  written  comments  to  our  survey  that  competition  for 
participants  and  duplication  of  services  due  to  lack  of  coordination  with 


^*This  issue  was  also  raised  in  a  study  that  found  that  the  continued  emphasis  on  measuring 
performance  in  categorical  programs  undermines  the  integration  expected  in  WIA.  Evelyn 
Ganzglass,  Martin  Jensen,  Neil  Ridley,  Martin  Simon,  and  Chris  Thompson.  Transforming 
State  Workforce  Development  Systems:  Case  Studies  of  Five  Leading  States  (National 
Governors’  Association,  2001). 
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other  programs  would  continue  as  long  as  each  program  is  required  to 
meet  its  own  performance  and  participation  levels.  Fourteen  states 
volunteered  in  their  written  comments  to  our  survey  that  the  federal 
government  should  work  to  coordinate  performance  measures  across 
programs  or  develop  systemwide  measures.  In  addition,  while  states  agree 
that  systemwide  measures  are  needed,  they  caution  against  making  any 
additional  measures  mandatory,  since  states  are  still  adapting  to  the 
existing  measures. 


Limited  Progress  on 
Developing  Performance 
Measures  for  One-Stop 
Systems 


Although  WIA  did  not  establish  one-stop  measures,  it  does  require  that 
Labor  develop  additional  optional  measures  to  assist  states  in  assessing 
progress  toward  their  workforce  investment  goals,^'*  which  Labor  has 
interpreted  to  include  one-stop  measures.  Labor  began  developing 
workforce  development  performance  measures  to  capture  overall  one- 
stop  use  after  one-stop  systems  were  piloted.  Since  the  passage  of  WIA, 
Labor  has  continued  its  efforts  to  develop  systemwide  measures,  but  it  has 
made  limited  progress.  Labor  convened  a  working  group  in  September 
2001  to  develop  additional  indicators  of  one-stop  performance.  Partner 
representatives  at  this  group  included  national  workforce-related 
organizations  including  the  National  Association  of  State  Workforce 
Agencies,  National  Governors’  Association,  National  Association  of 
Counties,  U.S.  Conference  of  Mayors,  and  representatives  of  states  and 
regional  boards.  This  group  is  working  to  develop  a  menu  of  indicators 
that  will  help  provide  a  comprehensive  picture  of  WIA  system  activity. 
Such  measures  may  include  capturing  information  on  self-service 
customers  and  the  cost  of  services  at  the  one-stop.^"  These  measures, 
while  optional,  would  help  provide  information  on  overall  one-stop  use 
across  the  country  if  all  states  report  on  at  least  some  of  the  measures. 
Labor  plans  to  have  guidelines  for  these  optional  indicators  in  place  for 
use  in  program  year  2002,  which  begins  on  July  1,  2002.  In  order  for  states 
to  be  able  to  implement  them  for  the  coming  program  year.  Labor  will 
need  to  provide  final  guidance  well  before  July  2002. 


^^Section  136  (i)(l)  requires  Labor  to  work  with  workforce  development  system  partners  to 
develop  terms  for  a  menu  of  additional  indicators  of  performance  to  help  states  assess 
their  progress  toward  their  workforce  investment  goals.  The  purpose  of  developing  these 
additional  measures  is  to  ensure  nationwide  comparability. 

^°Labor  is  currently  seeking  OMB  approval  to  collect  data  on  the  costs  and  usage  of  WIA 
and  Wagner-Peyser  funded  services  that  do  not  require  registration.  Labor  issued  a  Federal 
Register  Notice  on  January  16,  2002  seeking  public  comment  on  this  proposal. 
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In  addition  to  these  national  efforts,  some  states,  on  their  own  initiative, 
have  attempted  to  develop  additional  measures  for  one-stop  systems,  but 
these  efforts  are  not  coordinated  and  do  not  allow  for  nationwide 
assessment  of  the  one-stop  system.  According  to  our  survey,  eight  states 
have  created  or  are  developing  additional  systemwide  measures,  but  of 
those,  only  three  are  reporting  them  to  Labor.  Pennsylvania,  for  example, 
developed  five  measures  specific  to  its  one-stop  system’s  performance. 
These  indicators,  intended  to  measure  the  overall  effectiveness  of  the  one- 
stop  system,  include  median  cycle  time  to  fill  a  job,  and  the  percentage  of 
employers  and  individuals  using  services  through  the  one-stop.  Florida,  on 
the  other  hand,  has  developed  “tiers”  of  measures  that  focus  on  the 
outcome  of  their  workforce  development  programs.  In  the  first  tier,  state¬ 
generated  systemwide  indicators  measure  many  employment  and  training 
programs  together.  The  second  tier  clusters  similar  types  of  programs  and 
captures  measures  relevant  to  particular  groups  (e.g.,  continued  education 
status  of  youth  in  youth  programs).  The  third  tier  captures  all  the  federally 
mandated  measurements,  as  well  as  measures  for  the  other  tiers,  such  as 
caseloads  for  specific  programs.  In  this  way,  Florida  has  attempted  to 
measure  the  system  overall  as  well  as  outcomes  for  individual  programs. 
Measures  developed  by  other  states  include  the  number  of  people  using 
the  resource  rooms  at  one-stops  and  the  increase  over  time  in  the  number 
of  unemployed  people  getting  a  job.  Despite  these  states’  efforts,  the 
absence  of  nationally  established  systemwide  measures  means  that  Labor 
cannot  ensure  nationwide  comparability. 


Conclusions 


WIA  represents  a  fundamental  shift  in  the  way  federally  funded 
employment  and  training  services  are  provided  to  job  seekers  and  in  the 
way  WIA  programs  measure  and  monitor  success.  Despite  obstacles,  in 
just  over  a  year  states  have  made  good  progress  in  implementing  the  new 
requirements  under  WIA — developing  new  processes  and  designing  new 
systems.  Labor,  for  its  part,  has  been  working  to  find  ways  to  allow  states 
and  localities  greater  flexibility  to  design  their  programs  to  meet  local 
needs  and  has  been  actively  seeking  opportunities  for  states  to  have  input 
into  the  process,  particularly  in  the  area  of  performance  measurement.  But 
given  the  challenges  states  have  faced  in  implementing  the  new 
performance  measurement  system,  more  time  is  needed  before  the 
measures  can  meaningfully  gauge  the  success  of  the  programs. 

This  new  performance  measurement  system  under  WIA  is  a  high-stakes 
game — a  state’s  future  funding  and,  therefore,  its  ability  to  serve  its 
citizens  may  depend  upon  how  well  it  performs  compared  to  how  well  it  is 
expected  to  perform.  It  should  be  no  surprise  that  states  and  localities  are 
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designing  their  systems  and  processes  in  ways  that  will  enhance  their 
ability  to  meet  their  performance  levels.  Because  states  see  the  current 
performance  levels  as  too  high  for  the  current  economy,  states  and 
localities  may  choose  not  to  serve  those  job  seekers  who  may  be  helped  by 
their  services,  but  who  may  not  help  in  achieving  their  negotiated 
performance  levels.  Unless  the  performance  levels  can  be  adjusted  to  truly 
reflect  differences  in  economic  conditions  and  the  population  served,  local 
areas  will  continue  to  have  a  disincentive  to  serve  some  job  seekers  that 
could  be  helped. 

WIA’s  requirement  to  use  U1  data  to  track  outcomes  is  a  step  in  the  right 
direction — it  provides  federal,  state,  and  local  government  entities  with  an 
objective  means  to  evaluate  program  success.  But  it  brings  challenges  that 
need  to  be  addressed,  and  states  will  need  help  to  do  so.  Establishing  the 
means  to  routinely  share  data  across  state  lines  through  WRIS  and 
developing  ways  to  share  promising  approaches  in  the  use  of 
supplemental  data  sources  and  in  managing  the  assessment  of  short-term 
program  needs  would  go  far  in  moderating  these  challenges.  Without  this 
help  and  the  cooperative  efforts  of  states  and  localities  toward  this  end, 
developing  a  useful  performance  measurement  system  will  take  longer  and 
cost  more. 

In  general,  WIA’s  performance  measurement  system  captures  some  useful 
information,  but  it  may  not  capture  all  the  right  information.  The  measure 
to  track  credentials  has  limited  value  because  it  lacks  a  standard  definition 
for  what’s  being  measured.  For  other  measures,  the  lack  of  clear 
definitions  for  whom  to  track  limits  their  usefulness  in  drawing 
conclusions  about  program  success  at  both  the  state  and  national  levels. 
Without  clear  definitions  and  processes,  the  measures  will  not  provide  the 
Congress  with  a  true  picture  of  how  well  the  programs  are  performing. 
Furthermore,  WIA  performance  measures  gauge  only  WIA-funded 
services;  yet  there  is  widespread  agreement  that  measures  are  needed  to 
gauge  the  effectiveness  of  the  entire  one-stop  system.  The  system’s  narrow 
focus  on  program  outcomes  for  a  limited  number  of  participants  misses  a 
key  requirement  of  WIA  to  support  the  movement  toward  a  coordinated 
system.  In  fact,  the  measures  may  foster  the  opposite — a  siloed  approach 
that  encourages  competition  among  programs  and  limits  their 
cooperation.  Without  global  one-stop  measures,  the  Congress  will  not  be 
able  to  assess  how  well  states  and  localities  are  doing  in  meeting  WIA’s 
requirement  to  coordinate  services.  The  lack  of  such  measures  may, 
instead,  send  a  signal  to  states  that  service  coordination  is  a  minor  goal. 
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Recommendations  for 
Executive  Action 


To  give  states  and  local  areas  more  time  to  implement  WIA  performance 
measures  and  establish  baseline  data  needed  to  determine  performance 
levels,  we  recommend  that  the  Department  of  Labor 

•  delay  the  application  of  financial  sanctions  for  at  least  1  year  or  until  it 
is  judged  that  states  have  their  data  systems  sufficiently  in  place  to 
successfully  track  WIA  outcomes. 

To  eliminate  possible  disincentives  to  serve  some  job  seekers  and  ensure 
that  states  and  local  areas  will  not  be  unduly  penalized  for  economic 
downturns,  we  recommend  that  the  Department  of  Labor 

•  expedite  the  release  of  guidance  on  revising  negotiated  performance 
levels  and  allow  states  to  immediately  begin  the  process  of  re¬ 
negotiation. 

To  ensure  uniformity  in  data  collection  and  reporting  so  that  performance 
results  are  more  accurate  and  comparable  across  states,  we  recommend 
that  the  Department  of  Labor 

•  provide  clearer  guidance  using  objective  criteria  on  who  should  or 
should  not  be  registered  as  a  WIA  participant  for  tracking  purposes 
and,  once  the  guidance  is  released,  work  proactively  with  states  to 
implement  it,  and 

•  issue  guidance  delineating  a  clear  definition  for  what  constitutes  a 
credential,  and,  once  the  guidance  is  released,  ensure  that  states  use  it 
to  report  on  this  indicator. 

To  help  states  address  the  challenges  of  using  U1  data  to  measure 
outcomes,  we  recommend  that  the  Department  of  Labor 

•  continue  to  fully  fund  the  Wage  Record  Interchange  System  in  order  to 
facilitate  the  sharing  of  U1  data  across  state  lines;  and 

•  develop  ways  for  states  to  share  promising  approaches  in  the  use  of 
supplemental  data  sources  in  closing  the  data  gaps  for  covered  and 
uncovered  employment  in  Ul;  and 

•  develop  ways  for  states  to  share  promising  approaches  that  help  states 
address  the  Ul  timeliness  issue,  providing  methods  to  help  states 
monitor  and  improve  their  programs  in  a  timely  manner. 
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Agency  Comments 


To  help  states  measure  one-stop  performance,  we  recommend  that  the 
Department  of  Labor 

•  ensure  that  the  development  of  optional  one-stop  system  measures  is 
completed  in  enough  time  for  states  to  implement  them  at  the 
beginning  of  program  year  2002. 


We  provided  a  draft  of  this  report  to  Labor  for  its  review  and  comment. 
Labor’s  comments  are  in  appendix  11.  We  incorporated  comments  and 
clarifications  where  appropriate. 

Labor  generally  agreed  with  our  findings  and  recommendations,  noting 
that  they  are  consistent  with  information  they  have  gathered  from  state 
and  local  partners.  In  its  comments.  Labor  expressed  concern  that 
negotiated  performance  levels  may  be  determining  who  receives  WIA- 
funded  services,  indicating  that  it  will  work  with  states  and  local  areas  to 
address  this  issue.  Labor  also  commented  on  our  finding  regarding  the 
lack  of  clear  guidance  on  certain  policies,  stressing  the  importance  of  state 
and  local  flexibility  in  determining  specific  policies  and  practices  to  fit 
local  needs.  While  state  and  local  flexibility  is  important,  we  continue  to 
be  concerned  that  the  lack  of  a  uniform  understanding  of  when 
registration  occurs  and  what  constitutes  a  credential  raises  questions 
about  both  the  accuracy  and  the  comparability  of  states’  performance 
data.  We  are  pleased  to  note  that  Labor  is  in  the  process  of  reviewing  this 
issue.  Finally,  Labor  cites  its  efforts  to  collaborate  with  states  and  local 
areas  in  developing  a  performance  accountability  system  and  increasing 
partnerships.  We  commend  Labor  for  obtaining  states’  input  and 
participation  in  developing  such  a  system. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Labor,  relevant 
congressional  committees,  and  others  who  are  interested.  Copies  will  also 
be  made  available  to  others  upon  request.  The  report  is  also  available  on 
GAO’s  home  page  at  http://www.gao.gov. 
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Please  contact  me  on  (202)  512-7215  if  you  or  your  staff  have  any 
questions  about  this  report.  Other  major  contributors  to  this  report  are 
listed  in  appendix  111. 


Sigurd  R.  Nilsen 
Director,  Education,  Workforce, 
and  Income  Security  Issues 
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Appendix  I:  WIA  Performance  Measures  for 
Adults,  Dislocated  Workers,  and  Older  Youth 

The  following  table  includes  descriptions  of  the  performance  measures 
reviewed  in  this  report.  It  does  not  include  the  calculations  required  to 
attain  the  final  value  of  the  performance  measures. 

Adults 

Definition 

Entered  employment  rate 

Of  those  who  did  not  have  a  job  when  they  registered  for  WIA,  the  percentage  of  adults 
who  aot  a  iob  bv  the  end  of  the  I”*  quarter  after  exit.  This  measure  excludes  participants 
who  are  employed  at  the  time  of  registration. 

Employment  retention  rate  at  6  months 

Of  those  who  had  a  job  in  the  f‘  quarter  after  exit,  the  percentage  of  adults  who  have  a  job 
in  the  S"'  quarter  after  exit. 

Average  earnings  change  in  6  months 

Of  those  who  had  a  job  in  the  f‘  quarter  after  exit,  the  post-program  earnings  increases  as 
compared  with  pre-program  earnings. 

Employment  and  credential  rate 

Of  those  adults  who  received  WIA  training  services,  the  percentage  who  were  employed  in 
the  f‘  quarter  after  exit  and  received  a  credential  by  the  end  of  the  S"*  quarter  after  exit. 

Dislocated  workers 

Entered  employment  rate 

The  percentage  of  dislocated  workers  who  got  a  job  by  the  end  of  the  f  quarter  after  exit. 
This  measure  includes  dislocated  workers  who  are  employed  at  the  time  of  registration. 

Employment  retention  rate  at  6  months 

Of  those  who  had  a  job  in  the  T'  quarter  after  exit,  the  percentage  of  dislocated  workers 
who  have  a  job  in  the  3'“'  quarter  after  exit. 

Earnings  replacement  rate  in  6  months 

Of  those  who  had  a  job  in  the  T'  quarter  after  exit,  the  percentage  of  pre-program  earnings 
being  earned  post-program.  Since  it  may  be  difficult  to  find  dislocated  workers  jobs  with 
equivalent  or  better  wages,  this  measure  captures  the  percentage  of  earnings  of  the  new 
job  in  relation  to  the  old. 

Employment  and  credential  rate 

Of  those  dislocated  workers  who  received  WIA  training  services,  the  percentage  who  were 
employed  in  the  I”*  quarter  after  exit  and  received  a  credential  by  the  end  of  the  3"*  quarter 
after  exit. 

Older  youth  (age  19-21) 

Entered  employment  rate 

Of  those  who  are  not  employed  at  registration  and  who  are  not  enrolled  in  post-secondary 
education  or  advanced  training  in  the  T*  quarter  after  exit,  the  percentage  of  older  youth 
who  have  gotten  a  job  by  the  end  of  the  I”*  quarter  after  exit.  This  measure  also  excludes 
youth  that  move  on  to  post-secondary  education  or  advanced  training  and  not 
employment. 

Employment  retention  rate  at  6  months 

Of  those  who  are  employed  in  the  I”*  quarter  after  exit  and  who  are  not  enrolled  in  post¬ 
secondary  education  or  advanced  training  in  the  3'“  quarter  after  exit,  the  percentage  of 
older  youth  that  are  employed  in  the  3"*  quarter  after  exit. 

Average  earnings  change  in  6  months 

Of  those  who  had  a  job  in  the  1”*  quarter  after  exit  and  who  are  not  enrolled  in  post¬ 
secondary  education  or  advanced  training,  the  post-program  earnings  increases  as 
compared  with  pre-program  earnings. 

Employment/education/training  and 
credential  rate 

The  percentage  of  older  youth  who  are  in  employment,  post-secondary  education,  or 
advanced  training  in  the  T'  quarter  after  exit  and  received  a  credential  by  the  end  of  the  3'“' 
quarter  after  exit. 

Customer  satisfaction” 

Employer  customer  satisfaction 

The  average  of  three  statewide  survey  questions  rated  1-10:  (1  being  “very  dissatisfied” 
and  10  being  “very  satisfied”): 

•  was  the  employer  satisfied  with  services 
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Appendix  I:  WIA  Performance  Measnres  for 
Adnlts,  Dislocated  Workers,  and  Older  Youth 


Adults 

Definition 

•  did  the  service  meet  the  expectations  of  the  customer 

•  how  well  did  the  service  compare  to  the  ideal  set  of  services 

Participant  customer  satisfaction 

The  average  of  three  statewide  survey  questions  rated  1-10: 

•  was  the  participant  satisfied  with  services 

•  did  the  service  meet  the  expectations  of  the  customer 

•  how  well  did  the  service  compare  to  the  ideal  set  of  services 

"A  statewide  telephone  survey  of  a  sample  of  500  is  conducted  for  all  the  WlA-funded  programs. 


Source:  U.S.  Department  of  Labor  TEGL  7-99. 
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Appendix  II:  Comments  From  the  Department 
of  Labor 


U.S.  Department  of  Labor  Assistant  secretary  for 

Employment  and  Training 
Washington,  D.C.  20210 

JAN  !  6  2002 

Mr.  Sigurd  R.  Nilsen 
Director,  Education,  Workforce 
and  Income  Security  Issues 
U.S.  General  Accounting  Office 
Washington,  DC  20548 

Dear  Mr.  Nilsen: 

Thank  you  for  the  opportunity  to  provide  comments  on  the  draft  GAO  report. 
Workforce  Investment  Act:  Improvements  Needed  in  Performance  Measures  to  Provide 
More  Accurate  Picture  of  WlA's  Effectiveness  (GAO-02-275). 

In  general,  GAO's  findings  and  recommendations  are  consistent  with  what  we  have 
heard  in  discussions  of  the  performance  accountability  system  both  within  the 
Department  and  with  our  state  and  local  partners.  The  Department  of  Labor's 
approach  to  performance  accountability  under  title  1  of  the  Workforce  Investment  Act 
(WIA)  has  been  based  on:  (1)  the  system  change  created  by  WIA;  (2)  state  and  local 
flexibility;  and  (3)  increased  partnerships. 

The  System  Chanee  Created  by  WIA:  The  enactment  of  WIA  brought  dramatic  changes 
to  many  aspects  of  the  nation's  workforce  system.  Many  of  these  changes  were  in  the 
area  of  service  delivery.  WIA  streamlined  eligibility  and  service  strategy  procedures  in 
order  to  allow  individuals  to  be  served  in  an  expedited  and  customer-focused  manner. 
The  delivery  of  services  fo  customers  is  the  first  priority,  and  administrative  processes 
such  as  reporting  and  paperwork  become  less  important  than  program  performance. 
The  discussion  on  pages  14  and  15  of  the  draft  related  to  states  reporting  that 
performance  levels  may  determine  who  receives  WIA-funded  services  causes  us 
concern  that  not  all  areas  are  taking  advantage  of  the  opportunities  that  WIA  provides. 
In  2002,  ETA  staff  will  work  closely  with  states  and  local  areas  to  clarify  any 
misconceptions  in  this  crucial  area  and  to  stress  that  the  priority  of  the  workforce 
investment  system  is  to  provide  quality  services  to  customers. 

With  the  focus  on  service  delivery,  performance  accountability  becomes  even  more 
important.  For  example,  WIA  contains  sets  of  core  indicators  designed  to  measure  the 
youth,  adult,  and  dislocated  worker  programs,  as  well  as  customer  satisfaction 
indicators.  While  we  agree  that  measures  which  assess  the  performance  of  the  One- 
Stop  system  as  a  whole  are  needed,  WIA  does  not  contain  comprehensive  measures  for 
the  One-Stop  delivery  system.  As  noted  in  your  report,  last  September  we  began  to 
gather  information  and  recommendations  on  system  measures  to  support  state  use  of 
additional  performance  indicators  in  preparation  for  WIA  reauthorization  discussions. 

S  A  Proud  Member  of  America’s  Workforce  Network 
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Appendix  II:  Comments  from  the  Department 
of  Labor 


State  and  Local  Flexibility:  In  implementing  WIA,  the  Department  has  developed 
policies  and  guidance  designed  to  support  the  key  WIA  reform  principle  of  increased 
state  and  local  flexibility.  Accordingly,  when  developing  the  regulations  and 
performance  accountability  guidelines,  in  many  instances  the  Department  set  broad 
parameters  or  definitions,  leaving  the  process  of  establishing  specific  policies  or 
definitions  up  to  the  states  and  local  areas.  Two  examples  of  this  flexibility  are  touched 
upon  in  the  draft  report:  (1)  the  point  of  registration  for  adults  and  dislocated  workers, 
and  (2)  the  definition  of  a  credential  for  performance  accountability  purposes.  As  you 
noted  in  your  report,  we  have  provided  guidance  on  establishing  a  point  of  registration 
and  on  the  states'  authority  to  define  credentials,  but  in  both  cases  the  needs  of  the  job 
seeker  and  employer  customers  at  the  state  and  local  level  were  given  priority  over  a 
one-size-fits-all  national  requirement.  Based  on  state  and  local  input,  the  Department  is 
in  the  process  of  reviewing  these  policy  decisions.  No  decision  regarding  revisions  to 
these  existing  policies  has  been  made  at  this  time. 

Increased  Partnership:  WIA  also  calls  for  increased  partnerships  between  all  levels  of 
the  workforce  system.  Accordingly,  the  WIA  title  I  performance  accountability  system 
has  been  developed  through  an  unprecedented  collaborative  process.  The  Department 
has  made  great  efforts  to  ensure  that  the  performance  accountability  system  for  title  I 
was  designed  in  partnership  with  states  and  local  areas.  A  series  of  consultation  papers 
were  published  for  public  comment  in  the  Federal  Register  begirming  in  March  1999. 
The  March  1999  consultation  paper  specifically  asked  for  comment  on  several  of  the 
issues  discussed  in  the  draft  report,  such  as  (1)  the  point  at  which  adult  and  youth 
registrants  are  counted  for  different  performance  measures;  (2)  the  use  of  wage  records 
for  performance  measurement  considering  availability,  completeness,  and  accuracy; 
and  (3)  identifying  possible  unintended  effects  resulting  from  definitions/ policies 
around  performance  measurements. 

The  public  was  also  provided  an  opportunity  to  comment  on  the  interim  final 
regulations,  published  in  April  1999,  which  covered  the  point  of  registration  issue  and 
the  performance  accountability  system.  The  Department  hosted  two  rounds  of  town 
hall  meetings  to  solicit  comments  from  the  public  on  all  issues  related  to  the 
implementation  of  WIA,  including  the  area  of  performance.  Proposed  WIA  title  1 
reporting  requirements  were  published  in  the  Federal  Register  for  comment,  and 
consultation  with  workforce  partners  continued  to  ensure  that  all  comments  and  points 
of  view  had  been  considered. 

We  are  impressed  by  the  progress  that  our  state  and  local  partners  have  made  in 
implementing  the  performance  accountability  provisions  of  WIA  title  1.  However,  due 
to  the  complexity  of  building  a  comprehensive  and  seamless  workforce  system,  we 
anticipate  that  the  performance  infrastructure  will  continue  to  develop  over  the  next 
several  years,  and  that  lessons  learned  from  states  and  focal  areas  will  be  used  to 
improve  the  system  as  it  evolves. 
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Appendix  II:  Comments  from  the  Department 
of  Labor 


We  have  enclosed  some  additional  specific  comments  on  the  GAO's  findings.  We  will 
provide  a  formal  response  to  GAO's  recommendations  for  executive  action  when  the 
report  is  final.  In  addition,  we  expect  that  the  issue  of  performance  accountability  will 
be  a  key  issue  analyzed  as  part  of  the  upcoming  WlA  reauthorization  discussions. 
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GAO’S  Mission 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accoimtability  of  the  federal  government  for  the 
American  people.  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  policy,  and  fimding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  is  through  the 
Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full-text  files  of 
current  reports  and  testimony  and  an  expanding  archive  of  older  products.  The 

Web  site  features  a  search  engine  to  help  you  locate  documents  using  key  words 
and  phrases.  You  can  print  these  documents  in  their  entirety,  including  charts  and 
other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail 
this  list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  "Subscribe  to  daily 
e-mail  alert  for  newly  released  products"  under  the  GAO  Reports  heading. 
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